Design the Payment Pipeline

Understanding the problem

Start with the fact that reorders every priority. In a read-heavy feed a stale answer is harmless; here it is money. If a payment is lost, a merchant is out real funds; if it runs twice, a buyer is charged twice. And the pipeline that has to prevent both spans services and outside banks that PayPal does not control and cannot roll back, so correctness comes first, speed second. The work also splits into two phases that look like one. Authorizing a payment reserves the money. Capturing it moves the money. The gap between those two is where the hardest question in the design lives: a call timed out, so did the money move?

The PayPal angle

The entire round is graded on whether money is conserved when something breaks, far more than on how many payments a second you can push. PayPal moves on the order of $1.79 trillion a year across tens of billions of payments. It runs a closed-loop wallet, where PayPal often holds both the buyer's and the merchant's side of a transaction. That scale is the color; the invariant is the point. Two moves carry the interview. Put idempotency on the write path before anyone asks, using PayPal's real PayPal-Request-Id header, so a retried payment settles once. And treat every money movement as a balanced double-entry write, so the books can never silently drift. Candidate reports describe the same tilt across the loop: reach for the ledger and the request id yourself, name the failure modes before the interviewer does, and the throughput math can wait. Treat the boxes below as the well-understood public shape of a payment pipeline rather than a map of PayPal's exact internals, and you can still defend every one.

Functional requirements

Keep the core small and let the interviewer widen it; the depth here is money moving correctly under failure, and product breadth only pulls focus. The chain is short. Authorize a payment, which reserves the funds. Capture it, which moves them. Settle it, which delivers the money to the merchant's bank. Record every step in a ledger that always balances, and when a payment has to be undone, reverse it cleanly. Two of these carry the round on purpose: the authorize-versus-capture split, and the correctness of the recorded money.

Non-functional requirements

Reason each requirement from what a payment is. Correctness leads and does not bend: money is never created or destroyed by the system, only moved between accounts, and the books balance to the cent every time. A money movement is atomic, so all of its ledger entries commit together or none do, and a balance is never left half-updated. The ledger is strongly consistent, because it is the record of real money. Availability still matters, and the peripheral parts, notifying a merchant or refreshing a cached balance, are allowed to lag. But the core money path chooses a correct answer over a fast one; when in doubt, the pipeline makes the client retry rather than risk a double-spend.

One data-type rule sits under all of it. Hold every amount as an integer count of the smallest currency unit, cents for dollars, never a floating-point number. Floats lose fractions of a cent to rounding, and a ledger that rounds is a ledger that is wrong. PayPal interviewers treat this as table stakes: money as integer cents or a fixed-precision decimal, and a candidate who reaches for a float has answered a different, easier question.

• Conserved Money. Debits equal credits, always.
• Exactly once Per request id. Retries never double-spend.
• Strong Ledger consistency. Correctness beats availability.
• Integer cents Money type. Never a float.

Sizing the pipeline

Put the numbers down, because they explain why correctness is affordable here. PayPal reports on the order of 25.4 billion payments a year. Spread across a year that is only about 800 payments a second on average, a gentle aggregate rate next to a read-heavy consumer feed. Spikes and skew threaten it more than raw volume. Holiday peaks like Black Friday and Cyber Monday run several times the average, so the pipeline is built for a few thousand payments a second, well above 800. And a single large merchant during a flash sale can aim a flood of captures at one account, the same hot-account problem the ledger has to survive. Treat every figure here as an illustrative teaching estimate, good to an order of magnitude.

• Volume: ~25.4B txns/yr. PayPal's reported annual payment count. Averaged over a year that is only about 800 payments a second, a modest aggregate rate. Illustrative for sizing.
• Peak: 3-10x on Black Friday. Holiday spikes multiply the average several times over, so the pipeline is provisioned for a few thousand payments a second rather than the mean.
• Payment volume: ~$1.79T/yr. Total payment value flowing through, up about 7% year over year. The dollar figure is why one lost or doubled cent is a real incident rather than a rounding footnote.
• Ledger writes: ~2-6 per payment. Each payment posts balanced double-entry rows across authorize, capture, settle, so the append-only ledger absorbs a few thousand small row inserts a second at peak.

Two things fall out of the arithmetic. First, strong consistency is cheap at this rate. A few thousand payments a second, each fanning into a handful of tiny ledger rows, fits well within one partitioned relational primary per shard. The design never has to trade correctness away for throughput. Second, the money at stake is enormous, around $1.79 trillion a year, which is exactly why a lost or doubled cent is an incident and not a footnote. The rest of the design spends its whole budget staying correct across that spike and that skew.

The set-up

Core entities

Name the objects the pipeline moves before drawing anything. Five carry the design, and the one to fix early is the payment itself, because it is a state machine, not a single row you flip to done.

The store under the money is Postgres, and the choice is load-bearing, so be ready to defend it. All the entries of one movement have to commit together and sum to zero before anything is visible. A single relational transaction gives you that: begin, insert the entries, check the invariant, commit. It also needs a unique key on PayPal-Request-Id so a retry cannot post twice, which a unique constraint enforces atomically. The rows are tiny and uniform, so the tables are enormous in count but trivial per row, and the ledger partitions by account as it grows. Alongside it, Kafka carries lifecycle events between stages, so the orchestrator can drive authorize, then capture, then settle without holding one long transaction across external calls. An eventually-consistent key-value store cannot promise a movement's entries land atomically and balance, and that promise is the one a ledger cannot live without.

API

The external surface is deliberately thin. A caller creates a payment with an idempotency header, captures it, then reverses it when needed. Every amount is integer cents on the wire and in storage, and there is no endpoint that edits or deletes a recorded movement, because that operation does not exist. Reference these by name in the walkthrough and the deep dives.

Two calls do the real work. POST /v2/payments authorizes and returns fast, placing a hold rather than moving money, and reusing its PayPal-Request-Id on a retry replays the original instead of authorizing a second time. POST /v2/payments/{id}/capture is where the money moves and the ledger gets posted, and it is separate on purpose, because for anything that ships before it bills, capture happens later than authorize. Refund exists as its own call because a correction is a new, compensating transaction, never an edit to a past one. The deep dives build each of these.

High-level design

Build it one stage at a time and name each bottleneck as it surfaces. The write path is a short durable record followed by an orchestrated lifecycle, and the money path is where the interesting work lives. The diagram colors each box by role. Blue is the Buyer and Merchant, and violet is the services. Amber marks the Event log and the External rails the pipeline calls but does not own, and green marks the stores. The walkthrough tags the same components.

1) Receive, deduplicate, then authorize. A Buyer pays a merchant through the API gateway, which captures the PayPal-Request-Id at the edge and checks it against the Idempotency store. If that id already ran, the gateway replays the stored response and stops, so a retried payment never authorizes twice. For a fresh request the Payment orchestrator takes over; it validates, calls the Risk gate inline on a tight budget, and selects the funding source. Then it asks the External rails to place a hold that reserves the money without moving it; the payment is now authorized. What is punted here is the scary part; that hold crosses a boundary PayPal cannot roll back, so a timed-out authorize needs a precise resume rule, which is deep dive two.

2) Capture and post to the ledger. When the Merchant captures, immediately for most checkouts or later for orders that ship first, the Payment orchestrator consumes the hold and moves the money, then calls the Ledger service to record it. The ledger writes a balanced double-entry transaction into the Ledger store in one atomic commit: a debit and an equal credit that sum to zero, in integer cents. The payment becomes captured. Partial capture and void live here too, since a merchant may capture less than was authorized or release the hold entirely. The punt: when many captures hit one hot merchant at once, that account must not serialize on a single balance row, which the ledger dive handles.

3) Settle the funds. Capturing moves the money inside PayPal; settlement is how it reaches the merchant's real bank. This runs in two modes. Movement that stays inside the closed-loop wallet, PayPal balance to PayPal balance, settles in near real time as ledger entries. Movement that goes out to a card network or a bank over ACH settles in end-of-day batches. The Payment orchestrator hands a settlement file to the External rails and later reconciles against their clearing files; the payment reaches settled only once the external side confirms. The trap named up front, and returned to in deep dive five, is settling the same movement twice.

4) Notify and reconcile. On every state change the Payment orchestrator emits an event to the Event log. That drives the merchant notification (delivered by the separate webhooks pipeline) and downstream bookkeeping, all at-least-once and idempotent, so a duplicate event is harmless. Off to the side and entirely off the money path, a reconciliation job streams settled movements out of the Ledger store and compares them line by line against the banks' and networks' settlement files. Any drift is corrected the only way the ledger allows, with a reversing transaction, never an edit. That safety net is the last deep dive.

Technology choices

Each box on the diagram is a real category of tool. Name a concrete pick, say why that category fits a payment pipeline specifically, and have the failure question ready, because a correctness round pushes on every dependency you put on the board.

• Ledger store is Postgres, and the choice is load-bearing rather than incidental. A ledger needs all the entries of one movement to commit together and to enforce the zero-sum invariant before anything is visible, which is what a single relational transaction gives you directly. It also needs a unique constraint on PayPal-Request-Id so a retry cannot double-post. The rows are tiny integers and a currency code, so the table is enormous in count but trivial per row, partitioned by account as it grows. The failure probe is the hot merchant: a single mutable balance row would serialize a flash sale on one lock. So the authoritative entries stay append-only, where inserts never contend, and the balance is striped across many rows and summed. An eventually-consistent store cannot promise a transaction's entries land atomically and balance, and that is the one guarantee this system cannot give up.
• Event log is Kafka, a partitioned durable log carrying a lifecycle event on every state change. A log fits because the pipeline is a sequence of stages that must not block each other; the orchestrator commits a state and emits an event. Notification, analytics and bookkeeping drain it at their own pace. Consumers replay from an offset, fall behind, and catch up without adding load to the money path. The probe is the dual-write trap, where the service writes Postgres and Kafka separately and one succeeds alone. The fix is the outbox pattern: write the event into the same database transaction as the ledger entry, so a movement is published only because it was already durably recorded.
• Payment orchestrator is a stateless service running the payment as a persisted state machine, with the lifecycle stored in Postgres rather than in memory. Stateless is the point: every invariant lives in the database, so any instance can advance any payment and you scale by adding boxes. The probe is what happens when an instance dies mid-transition. Each transition is durable before the next begins, and every step is idempotent. A restarted orchestrator reads the current state and resumes. If a step half-completed, it runs the compensating action instead of stranding the payment.
• Idempotency store is a Postgres table keyed on PayPal-Request-Id, one row per request, holding the fingerprint, recovery point, lock, and stored response. It is the dedup store that makes retries safe, and a background sweep prunes rows past the retry window so it stays bounded rather than growing forever. The probe is why not a fast key-value cache. Two replicas of an eventually-consistent store can each accept the same id and reconcile later, after both have already paid. That is the exact double-spend the header exists to prevent.
• External rails are the card networks and banks the pipeline reaches through a processor gateway, and they are foreign state mutations no database can roll back. So idempotency is pushed one layer down: the orchestrator passes its own token, so a resubmission after a crash authorizes the card once and returns the same answer. The probe is what if the rail does not honor the token. Then the orchestrator queries it for that token's status before deciding, and batch reconciliation against the settlement file is the net that catches any disagreement.

Potential deep dives

The rest of the grade is made here. The interviewer fixes on one property of the pipeline and pushes until it bends. Staying correct under load and failure is what the round tests. These five are where a payment pipeline leans, each scored against the tiers an interviewer grades you on.

1) The auth-capture-settle lifecycle: states, holds, one orchestrator

A payment is a sequence of steps across services and outside banks, and any of them can fail. Get past modeling it as a single call that either works or does not. Authorization reserves the money with a hold. Capture consumes the hold and moves it. Settlement delivers it to the merchant's bank. Between each step the money sits in a known state, and the design's job is to make every transition survive a crash. Work up to it in tiers.

The PayPal angle

PayPal runs a two-sided closed-loop wallet, so the orchestrator is the design's spine, not a detail. A one-sided online gateway sees only part of the flow. PayPal often holds the buyer's funding source, the merchant's balance, and the ledger between them, so one service really does own the payment from authorize all the way to settle. Naming the auth-capture-settle split, and the expiring hold in between, signals you have moved money that ships before it bills. It also answers a common generic prompt. A booking or reservation system with payment holds and no double-booking has this exact authorize-then-capture shape with an expiring hold, so the pattern travels well past PayPal.

2) Exactly-once under retries: the PayPal-Request-Id header

A client fires a payment, the server processes it, and the response is lost on the way back. A timeout. The client cannot tell a lost request from a lost response, so if it retries naively it pays twice, and if it gives up it may have paid for nothing. This single failure, a dropped response over an unreliable network, is what idempotency exists to solve. Exactly-once delivery over a network is impossible, so aim for the achievable pair. The client retries until it hears back, which is at-least-once delivery, and the server makes processing idempotent, so a repeat produces the same one result. Together that is effectively-once, the property a payment needs.

The PayPal angle

PayPal-Request-Id is a real, public PayPal API contract, so use it by name and use it early. PayPal documents a client-generated id, unique per request and per call type, that the server stores and replays on a retry after a 5xx or a timeout. It plays the same role an idempotency key plays on a card processor, and reaching for it before the interviewer asks is the clearest signal that you have shipped money code. Defend the strongly-consistent store while you are there; two replicas that each accept the same id independently is a double-spend waiting to happen. A unique constraint is the cheapest way to make that impossible. The webhooks design leans on the same idempotent-consumer idea on the delivery side.

3) The double-entry ledger: money conserved by construction

Underneath every capture and every refund sits one question: how do you record money movement so it can never silently drift? The answer is centuries old. Double-entry bookkeeping records each movement as balanced debits and credits that sum to zero, so money is only ever moved between accounts, never created or destroyed. Build the ledger this way and the money-conservation invariant holds by construction, without relying on every writer to do its arithmetic right.

4) Why money is strongly consistent, and how it stays consistent across services

A feed can be eventually consistent because a late post is harmless. Money cannot, because a balance that is wrong for even a moment can authorize a payment that should have been declined. So the ledger and the balances it guards are strongly consistent: a write is visible only once it is durable and the invariant holds. The hard part is that a payment touches several services and outside banks, and you cannot wrap all of them in one database transaction. The question becomes how to keep the money invariant while letting the pieces coordinate.

For a payment pipeline the saga wins, and the reason is the external rails. Two-phase commit needs every participant to join a prepare-then-commit handshake, and a card network simply does not offer one, so it can never sit inside the atomic bracket. A saga models what really happens; each step (authorize, then capture, then the ledger post) is local and durable on its own. If a later step fails, a compensating step (void the hold, reverse the entry, refund the capture) walks the money back. The single strong-consistency island is the ledger transaction itself, where the balanced entries commit atomically. Everything around it is coordinated rather than locked.

5) External rails, async settlement, and reversal under partial failure

The last stretch is the one PayPal does not fully control: getting money out to banks and card networks, and undoing a payment when it has to be undone. Settlement runs in two modes, and conflating them is a common miss. Movement that stays inside the closed-loop wallet settles in near real time, since it is just ledger entries between PayPal accounts. Movement that crosses to an outside bank over ACH or to a card network settles asynchronously, in end-of-day batch files that clear hours later. The design has to treat that outside leg as slow, unreliable, impossible to roll back.

Then handle failure on purpose rather than by accident. The pipeline names each failure mode and its response.

• A timed-out authorize resumes from the recovery point. The request record is durable before the rail is ever called. A retry reads how far the payment got, then either resubmits the idempotent authorize or queries the rail for the token's status. It never blindly re-charges.
• A capture that fails after the hold is consumed triggers the saga compensation. The Payment orchestrator voids or reverses the movement and returns the payment to a clean state, rather than leaving money in limbo between captured and settled.
• A slow risk gate degrades on a decision, not a hang. The inline fraud call runs against a tight budget, and if it misses that budget the orchestrator applies a fail-closed or fail-open policy on purpose rather than stalling the payment behind it.
• A duplicate settlement file or event is absorbed, not doubled. Settlement instructions and lifecycle events are idempotent on their ids, so replaying a batch or redelivering an event changes nothing, which is what makes at-least-once delivery safe.

The PayPal angle

Dual-mode settlement and reconciliation are where PayPal's closed-loop, 25-year-old scale shows. A wallet that holds both sides of many transactions can settle a large share of movement internally in near real time. Only the leg that leaves for a bank or card network waits on batch clearing. Name that split, name the trap the prompt is really testing, settling the same movement twice, and you show you understand where a payment pipeline breaks. The reconciliation pass earns its place here: it is how the books stay honest against an outside world PayPal instructs but does not own. The cross-border design pushes this same settle-across-rails story into multiple currencies.

What is expected at each level

That is a wide surface for one pipeline. How much you must land depends on the level you are interviewing for.

Mid-level engineer

Breadth leads, roughly 80/20. Reach one coherent flow: authorize places a hold, capture moves the money and posts a balanced double-entry transaction, and a retry carrying the same PayPal-Request-Id returns the original result instead of paying twice. Hold money in integer cents and explain why the books must balance. The settlement modes and the reversal state machine can stay the interviewer's to raise. The bar: a correct happy path plus one safe retry, even if the deeper failure cases need prompting.

Senior engineer

Depth enters, roughly 60/40, and this is the primary independent bar. Model the payment as an explicit state machine and reach the auth-capture-settle lifecycle with holds on your own. Put idempotency on the write path unprompted and defend the strongly-consistent store, including why an eventually-consistent one would let two retries both think they are first. Reach for double-entry with reversing corrections, and choose a saga over two-phase commit because the external rails cannot join a distributed transaction. The bar: a near-complete pipeline where you found the timed-out-authorize problem yourself and had a precise answer for it.

Staff and beyond

Depth leads across several fronts, roughly 40/60. Own the operational and consistency story; that covers dual-mode settlement with idempotent batches, reconciliation against clearing files, and the outbox pattern for events. Reversals (refund, ACH return, chargeback) are defined transitions from every state they can hit. Reason clearly about where strong consistency is required and where eventual consistency with guaranteed reconciliation is safe, and about the hot-merchant skew on the ledger. You drive while the interviewer redirects rather than rescues. The bar: you hold the lifecycle, the ledger invariant, plus the settlement-and-reversal story as distinct, defensible decisions, while keeping correctness intact end to end.